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which communicate and coordinate actions among them- out of order stream of packets and/or disassembled stream of 

selves. The reassembly components distribute information packets. Device 100 receives the packets and resequences 

as to received packets and coordinate the sending of packets and/or reassembles the packets according to the invention, 

from themselves so as to produce the reassembled larger In one embodiment, device 100 consumes the resequenced 

packets. In one embodiment, each of the multiple reassem- 5 and/or reassembled stream of packets. In one embodiment, 

bly components maintains one or more data structures device 100 sends the resequenced and/or reassembled 

indicating packets stored locally and those packets stored stream of packets out to destination 112. 

anywhere (or elsewhere) within the multiple reassembly FIGS. 1B-D illustrate the basic topology of different 

components. When all packets comprising a larger packet exemplary packet switching systems resequencing and/or 

are received by one of the distributed resequencing 10 reassembling packets according to the invention. FIG. IB 

components, the reassembly components transmit their illustrates an exemplary packet switch 115 having multiple 

packets typically over a common bus or link in a coordinated inputs and outputs and a single interconnection network 120. 

fashion as to produce the original larger packet. FIG. 1C illustrates an exemplary packet switch 140 having 

multiple interconnection networks 141 and folded input and 

Details of Exemplary Embodiments 15 outpu t interfaces 149. FIG. ID illustrates an exemplary 

FIGS. 1A-1D and their discussion herein are intended to f° lded P acket switch 160 havin g multiple interconnection 

provide a description of various exemplary embodiments networks 161 and folded input and output interfaces 169. 

and operating environments in which the invention may be Embodiments of each of these packet switches 115, 140 and 

practiced. FIG. 1A illustrates an embodiment including a 160 resequencing and/or reassembling packets according to 

computer system resequencing and/or reassembling packets 20 the invention in the manners disclosed herein along with all 

according to the invention. FIGS. 1B-D illustrate embodi- possible embodiments within the doctrine of equivalents. Of 

ments including various packet switching systems rese- course » tne invention is not limited to these illustrated 

quencing and/or reassembling packets according to the operating environments and embodiments, and the packet 

invention switching systems may have more or less elements. 

FIG. lAillustrates a block diagram of one embodiment of 25 FIGJBoJlustrates an exemplary embodiment of a packet ^ 
a device 100 with resequences or reassembles a received one switch^mSJ Packet switch 115 comprises multiple input / lj> 
or more streams of packets. Embodiments of device 100 interfaceni7, interconnection network 120, and output 
include a routers, computer systems, other communications interfaces 129. Input interfaces 117 and output interfaces 
devices, etc, or components thereof. As shown, device 100 10 129 are both coupled over multiple links to interconnection 
includes processor and/or control logic 102 (hereinafter network 120 Line cards n6 131 are cou P led to m P ut 
"processor"), memory 101, storage devices 104, network interfaces 117 and output interfaces 131. In some embodi- 
interface(s) 105, and one or more internal communications meats including other packet switching topologies, line 
mechanisms 103 (shown as a bus for illustrative purposes). cards or their functionality may be included in the packet 
In one embodiment, processor 102 controls the operations of 35 switch itself > or as P art of the P acket switching system, 
device 100 to resequence and/or reassemble packets accord- In one embodiment, interconnection network 120 com- 
ing to the invention. Memory 101 is one type of computer- prises multiple switch elements SE-1 122, SE-2 125, and 
readable medium, and typically comprises random access SE-3 128 that are interconnected by multiple links. Line 
memory (RAM), read only memory (ROM), integrated cards 116 and 131 may connect to other systems (not shown) 
circuits, and/or other memory components. Memory 101 4Q to provide data items (e.g., packets) to be routed by packet 
typically stores computer-executable instructions to be switch 115. Although resequencing and reassembly of pack- 
executed by processor 102 and/or data which is manipulated ets can be accomplished in other components in accordance 
by processor 102 for implementing resequencing and/or with the invention, typically packets are resequenced and/or 
reassembly functionality in accordance with the invention. reassembled in output interfaces 129. 
Storage devices 104 are another type of computer-readable 45 FIG. 1C illustrates another exemplary operating environ- 
medium, and typically comprise disk drives, diskettes, net- me nt and embodiment of a packet switch 140. Packet switch 
worked services, tape drives, and other storage devices. 140 comprises multiple folded input and output interfaces 
Storage devices 104 typically store computer-executable 149 interconnected over multiple links to interconnection 
instructions to be executed by processor 102 and/or data networks 141, which are interconnected over multiple links 
which is manipulated by processor 102 for implementing 5Q returning to input and output interfaces 149. In one 
resequencing and/or reassembly functionality in accordance embodiment, interconnection networks 141 comprise mul- 
with the invention. tiple switch elements SE-1 142, SE-2 145, and SE-3 148 also 
In one embodiment, device 100 operates as a router, interconnected by multiple links. Interfaces 149 may con- 
bridge, switch or other communications device attached to a nect via bi-directional links to line cards 139 that connect 
communications network 107. Packets sent from source 108 55 with other systems (not shown) to provide data items (e.g., 
over link 109 to communications network 107 are relayed packets) to be routed by packet switch 140. Although 
over one or more links 106 to device 100. Device 100 resequencing and reassembly of packets can be accom- 
receives the packets, internally routes and resequences and/ plished in other components in accordance with the 
or reassembles the packets according to the invention. In one invention, typically packets are resequenced and/or reas- 
embodiment, device 100 consumes the resequenced and/or 6 o sembled in input/output interfaces 149. 
reassembled stream of packets. In one embodiment, device FIG, ID illustrates another exemplary operating environ- 
100 sends the resequenced and/or reassembled stream of ment and embodiment of a packet switch 160, Packet switch 
packets out one or more links 106 to destination 112, which 160 has a folded network topology. Packet switch 160 
receives these packets from communications network 107 comprises multiple folded input and output interfaces 169 
over link 111. 65 interconnected over multiple links to interconnection net- 
In one embodiment, device 100 operates as a computer or works 161, which are interconnected over multiple links 
communications system which receives from source 108 an returning to interfaces 169. In one embodiment, intercon- 
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FIG. 3 illustrates one embodiment of a mechanism for position pointer 444 which is updated using the current 

distributed resequencing and reassembly of packets in the sequence number. Ring buffer 442 could be implemented 

context of a packet switching system 300. In this exemplary using a linked list, array, or other data structure format. Ring 

embodiment, packets are distributed by a packet switch 301, buffer 442 has numerous buckets 443A-H (only eight are 

such as an eight plane packet switch, to four distributed 5 shown for illustration convenience) with the number of 

resequencing and reassembly components 303A-D. In one buckets typically related to the size of the out of order 

embodiment, each of two planes of the packet switch 301 are window. An example of an bucket entry 447 includes • 

connected to one of the distributed resequencing and reas- P acket ^ fi /^?/ nd ,? ayl °? d P T field f 9> w 

sembly components 303A-D over links 302A-H. Distrib- P acket £ e f der fi «" 448 ^11 contain the received packet 

uted resequencing and reassembly components 303A-D 10 head " fields, and the payload pointer points to the corre- 

communicate and coordinate the resequencing and reassem- sponding packet payload 449 stored in the packet memory 

bly operations over ring 304A-D. In a coordinated fashion, ™™f\ 42 ,°- ■J 0 ™* P acket headers are placed in 

distributed resequencing and reassembly components local data ^cture 440 at a bucket position offset from the 

303A-D send packets on packet merge bus 305B-E to curl ? nt P 0Sl . tl0 ° P 01D , ter *44 based on the sequence number 

produce the resequenced aid reassembled packets. The is of the rec ' ived P acket header and ^ Sf^ e number 

operation of a distributed resequencing and reassembly corresponding to the current position pointer 444 

components 303A-D are further described in relation to A global data structure 450 is used to identify packet 

FIGS 4A-D 5-8 headers which are stored in any of the distributed rese- 

' , '. ' , ,. , .. . , . , quencer and reassembly components 303A-D (or at least the 

une emoodiment ot disu-ioutea resequencmg ana reas- ^ distributed resequencer and reassembly components 

sembly component 303B (FIG. 3) is shown in FIG. 4A. 20 C _ D ag ^ ^ ystored packe , headers m identified 

Packets are received by the packet memory manager over . ^ ^ structure m) Global da , a sMK 45Q 

hnks 302C-D from the packet switch 301. An exemplary , , , , „ . „ , ,V, .... „ ... o „,„_, ' 

, A . . . , . , , ,. .„ . . . A ~n take the form of a nng buffer 452 with a current position 

format of such packets is ulustrated in FIG. A packet 430 d usi the ^ 

includes a cell header 431 and cell payload 432. The cell buffer ^ ^ ^ . lemented usi a 
header has various fields, mcludmg a packet type, source, 25 ^ ^ ^ ^ ^ 
destination abnc priority, sequence number packet length, ^ ^ buckets 453A _ R ( ^ m shown 
reserved field and cyclic reoundan^ for aiustration convenience) ^ the numb er of buckets 
correcting code value as illustrated instable 43 > IA^The fcaU fdated tQ ^ size of ±t QU{ of ordef In 
packet payload 432 may contain 3Q one embodiment, each of the buckets 453A-H contains a 
packets £g ■ 1 single 96 byte packet 432A or two 48 byte 30 corresponding packet 
packets 433-434) in a payload 4MB. Of course, FIG. 4B ^ ^ ^ £ Qf ^ fencer and 
illustrates one of numerous packet formats which may be in reassembl components 30 3A-D (or at least the other dis- 
embodiments of the invention. tributed resequencer and rea ssembly components 303A, 

Packet memory manager 420 maintains the packet pay- C-D). 

loads and sends the received packet headers to the packet 35 packet resequencer 402 coordinates its activities with the 

resequencer 402 over link 419. In addition, packet memory ket resequencers via the communication ring 304B, 404, 

manager 420 receives a data structure representing a reas- 3Q4 C, and packet reassembler 410 communicates with the 

sembled packet from packet reassembler 410 over link 418. Qther ket reassembler over this ring 30 4B, 404, 304C. 

Packet memory manager then retrieves from memory any periodically, packet sequencer 402 sends global update 

packet payloads stored locally corresponding to the reas- information to the other packet resequencers to identify the 

sembled packet. Each of the distributed resequencing and ket headers stored locally Referencing the local and 

reassembly components 303A-D places packets on the global data structures ^ 450 (FIG s. 4C _ D) t0 determine 

packet merge bus 305B-305E to generate the reassembled what kets nave been reC eived by the distributed rese- 

packet, which is sent out packet merge bus 305E to another ^ quencing and rea ssembly components 303A-D (FIG. 3), 

component or device. packet resequencer 402 can produce a stream of resequenced 

The operation of one embodiment of packet memory packet headers over link 405 to packet reassembler 410. 

manager 420 is illustrated in FIG. 7. Incoming packets are 0 ne embodiment of packet resequencer 402 is further 

received on Unks 302C-D and placed in incoming packet described in relation to FIG. 5. The operation of packet 

queues 713. Packets are then removed from incoming packet 5Q reS equencer 402 is controlled by control logic 510 which is 

queues 713 and sent to the packet data memory controller interconnected with other elements via communications link 

715. The packet payloads are then stored in packet data 5 U. Embodiments of communications link 511 include most 

memory 717. The packet headers are simultaneously sent by any communications mechanism, such as, but not limited to 

packet data memory controller 715 over link 419 to packet a buSj mlly connec ted mesh, point-to-point links, etc. Con- 

resequencer (FIG. 4A). The operation of the other elements 55 trol logic 510 process ce ll resequencing, stores and com- 

of packet memory manager 420 will be described hereinafter putes new res equencing state information based on infor- 

in relation to the packet merge process. mal i on updates received from other distributed resequencer 

Packet resequencer 402 receives these packet headers and and reassembly components 303A,C-D, and sends updates 

operates in conjunction with the packet resequencers of the to other distributed resequencer and reassembly components 

other distributed resequencing and reassembly components 60 303A,C-D. 

303A,C-D. In one embodiment, packet resequencer 402 Update messages representing the packets stored in the 

uses a local and a global data structures to resequence other distributed resequencer and reassembly components 

packets. 303A,C-D are received over ring 304B and placed in input 

FIG. 4C illustrates one embodiment of these data struc- queue 502, and outgoing update messages are placed in 

hires. A local data structure 440 is used to identify packets 65 output queue 506 and sent out over link 404. The local and 

which are received and locally stored. Local data structure global data structures 440, 450 (FIGS. 4C-D) are updated 

440 may take the form of a ring buffer 442 with a current and stored in data structure cache RAM 514. In one 
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packet descriptor to outgoing packet queues 705. Reas- 10. The system of claim 9, further comprising a packet 

sembled and partially reassembled packets arrive at outgo- switching means, connected to the plurality of means for 

ing packet queues 705 on link 305B. Each packet has a resequencing the stream of packets, for distributing packets 

sequence number associated with it, and if a packet in of tne stream 0 f packets to the plurality of means for 

outgoing packet queues 705 has a sequence number with a 5 resequencing lhe stream of packe ts. 

lower value, it is sent out on link 305C before the incoming ^ m of ^ g ^ Qf ^ fof 

packet. Otherwise, the incoming packet is passed through. , J t _ . , , , , * 

y . .( i i j * * * v* u *u resequencing the stream or packets includes a local data 

In view of the many possible embodiments to which the M & . .... • j- i * 

principles of our invention may be applied, it will be structu [ e ™ ans f ° r maintaining an indication of the packets 

appreciated that the embodiments and aspects thereof 1Q stored locally within a particular means for resequencing the 

described herein with respect to the drawings/figures are stream of packets. 

only illustrative and should not be taken as limiting the 12. A method for resequencing a stream of ordered 

scope of the invention. For example and as would be packets performed by a particular component of a plurality 

apparent to one skilled in the art, many of the process block 0 f components, the method comprising: 

operations can be re-ordered to be performed before, after, receivifl at ^ Qne ket of ^ stream of Qrdered 

or substantially concurrent with other operations. Also, ° . * r 

many different forms of data structures could be used in P ac e ' 

various embodiments. The invention as described herein receiving an indication of a set of packets received in at 
y\ contemplates all such embodiments as may co me within the least one other component of the plurality of compo- 
te rl£^^ scope of the following claims and equivalents fthereof) nents; 

What is claimed is: 20 ^^1^™ one or more data structures of the set of 

1. A system for resequencing a stream of packets com- fe ^ ^ ^ Qne ^ onent of the 
prisms* 

f i-* c j- * t. * a • 4 u plurality of components and the received at least one 

a plurality of distributed resequencing components, each i.r.i_ . c j j 1* 

of the plurality of distributed resequencing components packet of the stream of ordered packets; 

including one or more data structures for maintaining 25 maintaining a current position indication; and 

an indication of packets of the stream of packets that sending a particular packet out as part of the resequenced 

are stored in other distributed resequencing compo- stream based on a value of the current position indica- 

nents of the plurality of distributed resequencing com- t | on 

ponents; 13 ^he method of claim 12, further comprising updating 

a communications mechanism coupled to the plurality of 30 tne current position indication based on the sending of the 

distributed resequencing components to allow commu- particular packet 

nication among the plurality of distributed resequenc- 14 ^ method of daim 12> further comprising updatiDg 

ing components, an me curreot p 0S jti 0 n indication based on the received indi- 

one or more packet merging mechanisms coupled to the cationofthese , 0 f packets contained in the at least one other 

plurality of distributed resequencing components to 35 nent of th luralit f nents . 

receive packets of the stream or packets to produce a . J ™ . , r c , . , . 4l _ c 4U 

i , r * u * c i * 15. The method of claim 12, wherem the sending of the 

resequenced stream of the stream of packets. . , , . t , . . . . r ° , 

2. The system of claim 1, further comprising a distributor P^ 1 ^ P acket ] nchx6 f coordina ing the timing for send- 
for distributing the stream of packets to the plurality of m S of the Particular packet with at least another component 
distributed resequencing components. 40 of the plurality of components. 

3. The system of claim 2, wherein the distributor includes 16 - ^ method of claim 12, wherein the indication of the 
a plurality of paths through a packet switching system. set of packets contained in the at least one other component 

4. The system of claim 3, wherein at least two of the of the plurality of components is received over a commu- 
plurality of path are on different planes of the packet nications ring or bus. 

switching system. 45 17. The method of claim 12, further comprising sending 

5. The system of claim 1, wherein the communications an update indication identifying the sent particular packet to 
mechanism includes a communication ring or bus. one or more of the other components of the plurality of 

6. The system of claim 1, wherein the one or more packet components. 

merging mechanisms includes a packet merge bus. 18. The method of claim 17, wherein the update indication 

7. A packet switching system including the system of 50 mc i uc ies a sequence identifier of the sent particular packet, 
claim 1. 19. The method of claim 12, wherein each of the packets 

8. A router including the system of claim 1. of the stream of or d e red packets is identified with a sequence 

9. A system for resequencing a stream of packets com- identifier. 

prising: 20. The method of claim 19, wherein the sequence iden- 

a plurality of means for resequencing the stream of 55 tifier ^ a sequence num ber or a timestamp value, 

packets, each of the plurality of means for resequencing 21. The method of claim 12, further comprising storing 

the stream of packets including a data structure means the received at least one packet on a storage device or in a 

for indicating which packets of the stream of packets memory; and retrieving the received at least one packet from 

are located in other of the plurality of means for th e storage device or the memory. 

resequencing the stream of packets; 60 22. A packet switching system performing the method of 

a communications means coupled to the plurality of claim 12. 

means for resequencing the stream of packets; and 23. A router performing the method of claim 12. 

a merging means for receiving packets from the plurality 24. A computer system performing the method of claim 

of means for resequencing the stream of packets and for 12. 



producing a resequenced stream of the stream of pack- 
ets. 
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packets according to the invention in the manners disclosed herein along with all possible 
embodiments within the doctrine of equivalents. Of course, the invention is not limited 
to these illustrated operating environments and embodiments, and the packet switching 
systems may have more or less elements. 
5 FIG. IB illustrates an exemplary embodiment of a packet switcr^H^ Packet 

switch 1 15 comprises multiple input interfaces 1 17, interconnection network 120, and 
output interfaces 129. Input interfaces 117 and output interfaces 129 are both coupled 
over multiple links to interconnection network 120. Line cards 1 16 and 131 are coupled 
to input interfaces 1 17 and output interfaces 131. In some embodiments including other 

1 0 packet switching topologies, line cards or their functionality may be included in the 
packet switch itself, or as part of the packet switching system. 

In one embodiment, interconnection network 120 comprises multiple switch 
elements SE-1 122, SE-2 125, and SE-3 128 that are interconnected by multiple links. 
Line cards 1 16 and 13 1 may connect to other systems (not shown) to provide data items 

1 5 (e.g., packets) to be routed by packet switch 115. Although resequencing and reassembly 
of packets can be accomplished in other components in accordance with the invention, 
typically packets are resequenced and/or reassembled in output interfaces 129. 

FIG. 1C illustrates another exemplary operating environment and embodiment of 
a packet switch 140. Packet switch 140 comprises multiple folded input and output 

20 interfaces 149 interconnected over multiple links to interconnection networks 141, which 
are interconnected over multiple links returning to input and output interfaces 149. In one 
embodiment, interconnection networks 141 comprise multiple switch elements SE-1 142, 
SE-2 145, and SE-3 148 also interconnected by multiple links. Interfaces 149 may 
connect via bi-directional links to line cards 139 that connect with other systems (not 

25 shown) to provide data items (e.g., packets) to be routed by packet switch 140. Although 
resequencing and reassembly of packets can be accomplished in other components in 
accordance with the invention, typically packets are resequenced and/or reassembled in 
input/output interfaces 149. 
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to one of the distributed resequencing and reassembly components 3 03 A-D over links 
302A-H. Distributed resequencing and reassembly components 303A-D communicate 
and coordinate the resequencing and reassembly operations over ring 304A-D. In a 
coordinated fashion, distributed resequencing and reassembly components 303A-D send 
5 packets on packet merge bus 305B-E to produce the resequenced and reassembled 
packets. The operation of a distributed resequencing and reassembly 
components 303A-D are further described in relation to FIGs. 4A-D, 5-8. 

One embodiment of distributed resequencing and reassembly component 303B 
(FIG. 3) is shown in FIG. 4A. Packets are received by the packet memory manager over 
10 links 302C-D from the packet switch 301. An exemplary format of such packets is 
illustrated in FIG. 4B. A packet 430 includes a cell header 43 1 and cell payload 432. 
The cell header has various fields, including a packet type, source, destination, fabric 
priority, sequence number, packet length, res erved fie ld, and cyclic redundancy check or 
error-correcting code value as illustrated in gble 43 1A) The packet payload 432 may 
contain part or all of one or two packets (e.g., a single 96 byte packet 432A or two 48 
byte packets 433-434) in a payload 432B. Of course, FIG. 4B illustrates one of numerous 
packet formats which may be in embodiments of the invention. 

Packet memory manager 420 maintains the packet payloads and sends the 
received packet headers to the packet resequencer 402 over link 419. In addition, packet 
memory manager 420 receives a data structure representing a reassembled packet from 
packet reassembler 410 over link 418. Packet memory manager then retrieves from 
memory any packet payloads stored locally corresponding to the reassembled packet. 
Each of the distributed resequencing and reassembly components 303 A-D places packets 
on the packet merge bus 305B-305E to generate the reassembled packet, which is sent out 
25 packet merge bus 305E to another component or device. 

The operation of one embodiment of packet memory manager 420 is illustrated in 
FIG. 7. Incoming packets are received on links 302C-D and placed in incoming packet 
queues 713. Packets are then removed from incoming packet queues 713 and sent to the 
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With regards to FIG. 7, packets arrive on links 302 C-D and are temporarily stored 
in incoming packet queues 713. Packets are forwarded from incoming packet queues 713 
to packet data memory controller 715. Packet data memory controller 715 allocates 
memory for the packet, stores the packet in packet data memory 717, and forwards the 
5 packet header which contains a pointer to the packet to resequencing engine 402 (FIG. 5) 
over link 419. 

When packets are de-queued, a stream of packet descriptors arrive at packet 
merge queue 701 over link 418. Packet merge queue 701 forwards the packet pointers to 
packet data memory controller 71 5 which reads the packet out of packet data memory 717 

10 and forwards it to the outgoing packet queues 705. Packet merge queue 701 also 

forwards the packet descriptor to outgoing packet queues 705. Reassembled and partially 
reassembled packets arrive at outgoing packet queues 705 on link 305B. Each packet has 
a sequence number associated with it, and if a packet in outgoing packet queues 705 has a 
sequence number with a lower value, it is sent out on link 305C before the incoming 

15 packet. Otherwise, the incoming packet is passed through. 

In view of the many possible embodiments to which the principles of our 
invention may be applied, it will be appreciated that the embodiments and aspects thereof 
described herein with respect to the drawings/figures are only illustrative and should not 
be taken as limiting the scope of the invention. For example and as would be apparent to 

20 one skilled in the art, many of the process block operations can be re-ordered to be 

performed before, after, or substantially concurrent with other operations. Also, many 
different forms of data structures could be used in various embodiments. The invention 
as described herein contemplates all such embodiments as may come within the scope of 
the following claims and equivalent^hereof) 
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